Automated generation of early warning predictive insights about users

ABSTRACT

The present disclosure describes automated generation of early warning predictive insights derived from contextual analysis of user activity data of a distributed software platform. Predictive insights are automatically generated from analysis of user activity through implementation of trained artificial intelligence (AI) modeling. User activity data is accessed pertaining to user interactions by a plurality of users a software data platform. The trained AI modeling generates a plurality of mobility determinations that identify changes in patterns of user behavior over a current temporal filter associated with the user activity data. The plurality of mobility determinations is curated using business logic rules that evaluate a relevance of the mobility determinations. One or more predictive insights may be generated and presented via a graphical user interface notification. Exemplary notifications help provides insights into how user behavior has changed and why that is, thereby fostering understanding of predictions that can lead to actionable results.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This application is a non-provisional application that claims priorityto U.S. Provisional Patent Application No. 63/166,562 entitled“AUTOMATED GENERATION OF EARLY WARNING PREDICTIVE INSIGHTS ABOUT USERS”,filed on Mar. 26, 2021, which is hereby incorporated by referenceherein, in its entirety.

BACKGROUND

Traditional processing for early warning notifications focuses ontracking user progress in a course relative to completion of assignedcontent by students. Associated predictions are typically directed todetermining whether a user is on track to complete a course based onassignment completion and an associated grade. Typically, notificationsare raised only after a user missed an assignment or receives a lowgrade because those results may affect the larger goal of successfullycompleting a class/course. While those types of analytics are useful totrack the progress of a student relative to completion of a course as awhole, they do not provide a deeper understanding as to why a studentmay have missed an assignment or received a bad grade. As such, itstands that predictive outcome insights can be greatly improved byproviding a deeper understanding of user activity during onlinelearning.

Yet another technical challenge stems from understanding how to manageactivity data pertaining to user engagement with online learning tools.Activity data is voluminous and weakly correlated, making it extremelydifficult to generate meaningful data insights. Processing of such alarge amount of weakly correlated data requires significantcomputational resources, which would further need to be programmed forthat specific purpose of data insight generation. Challenges furtherarise when considering processing efficiency and latency issues, whereresults often need to be generated in real-time (or near real-time) tobe effective. This is especially true when trying to generate earlywarning notifications for educational progress.

Further technical challenges exist in the realm of e-learning.Traditional evaluation of e-learning activity offers a partial/sparseview of a student's overall learning activity. This makes it difficultgenerate contextually relevant user-specific insights for students andfurther frame insights in a way that other users (e.g., teachers,parents) can intervene to address a downward trend.

SUMMARY

For resolution of the above identified technical problems, as well asother reasons, there is a technical need for automated generation ofearly warning predictive insights derived from contextual analysis ofuser activity data of a distributed software platform. As a non-limitingexample, predictive insights are automatically generated from analysisof user activity data associated with an educational software platformthrough implementation of trained artificial intelligence (AI) modeling.For ease of understanding of the present disclosure, education is usedas an exemplary domain, but it is to be recognized that processing ofthe present disclosure is applicable to be customized for any type ofdomain. Continuing with education as an example domain, user activitydata is accessed pertaining to user interactions by a plurality of userswith the educational software platform. The trained AI modelinggenerates a plurality of mobility determinations that identify changesin patterns of user behavior over a current temporal filter associatedwith the user activity data. The plurality of mobility determinations iscurated based on an application of business logic rules that are used toevaluate a relevance of the mobility determinations. In some furtherinstances, an exemplary system is tuned to focus on specific types ofmobility determinations (e.g., downward trends and/or up-trendingmobility). For example, an exemplary system is calibrated to alert onteachers on potential drops in engagement of students. One or morepredictive insights may be generated and presented via a graphical userinterface (GUI) notification. Exemplary notifications help providesinsights into how user behavior has changed and why that is, therebyfostering understanding of predictions that can lead to actionableresults. In the age of hybrid/mixed learning, it has become more evidentthat teachers can no longer fully rely on direct interaction with thestudent to monitor a student's wellbeing and learning progress. As such,the present disclosure assists teachers by mining behavioral patterns toalert on potential drops in key indicators can provide an invaluablespotlight to focus a teacher's attention to students in need of support,possibly before their need escalates creating sustained lagging comparedto their peers.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Additionalaspects, features, and/or advantages of examples will be set forth inpart in the description which follows and, in part, will be apparentfrom the description, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive examples are described with reference tothe following figures.

FIG. 1A illustrates an exemplary system diagram of componentsinterfacing to enable automatic generation of mobility determinationsfrom which predictive data insights can be generated pertaining to usageof system/service, with which aspects of the present disclosure may bepracticed.

FIG. 1B illustrates an exemplary process flow for generating mobilitydeterminations and provision of predictive data insights to end users,with which aspects of the present disclosure may be practiced.

FIG. 1C illustrates an exemplary process flow for generating mobilitydeterminations within an educational software platform, with whichaspects of the present disclosure may be practiced.

FIG. 2 illustrates exemplary method related to automatic generation ofmobility determinations and management of predictive data insightsderived therefrom, with which aspects of the present disclosure may bepracticed.

FIGS. 3A-3C illustrate exemplary processing device views associated withuser interface examples for an improved user interface that is adaptedfor generation and management of predictive data insights frommanagement of mobility determinations, with which aspects of the presentdisclosure may be practiced.

FIG. 4 illustrates a computing system suitable for implementingprocessing operations described herein related to generation andmanagement of mobility determinations and derivation of predictive datainsights therefrom, with which aspects of the present disclosure may bepracticed.

DETAILED DESCRIPTION

As identified in the foregoing, there is a technical need for automatedgeneration of early warning predictive insights derived from contextualanalysis of user activity data of a distributed software platform. As anon-limiting example, predictive insights are automatically generatedfrom analysis of user activity data associated with an educationalsoftware platform through implementation of trained artificialintelligence (AI) modeling. For ease of understanding of the presentdisclosure, education is used as an exemplary domain, but it is to berecognized that processing of the present disclosure is applicable to becustomized for any type of domain.

Continuing with an education example, an educational platform enablesstudents, teachers, administrators, parents, etc., to be connectedthrough customized versions of software products that helps guidelearning (online, in-person and/or hybrid learning) and management ofassociated educational data. Integration of educational platformsprovide online learning resources and implementation of the same isknown to one skilled in the field of art. As a point of reference,capabilities of an educational platform comprise but are not limited to:registering students in courses; conducting online learning/classes;document storage, retention and management; documenting submission andgradings, transcripts, results of student tests and other assessments;management of access to content provided for learning; building studentschedules; tracking student attendance; managing data forextracurricular activities; and managing student-related data needs in aschool, among other examples. One example of a well-known software dataplatform is MICROSOFT365®, which may be tailored for management of aspecific document (e.g., education). In an educational domain example,applications/services of a software data platform (e.g., MICROSOFT365®)are tailored to further educational experiences for users. For instance,an online classroom and resources for management of associated users,content, etc., are enabled through MICROSOFT® Education therebyproviding an online learning (or hybrid learning) environment.

In one non-limiting example, user activity data is accessed pertainingto user interactions by a plurality of users with the educationalsoftware platform. A trained AI modeling generates a plurality ofmobility determinations that identify changes in patterns of userbehavior over a current temporal filter associated with the useractivity data. The plurality of mobility determinations is then curatedbased on an application of business logic rules that are used toevaluate a relevance of the mobility determinations. One or morepredictive insights may be generated and presented via a GUInotification. Exemplary notifications help provides insights into howuser behavior has changed and why that is, thereby fosteringunderstanding of predictions that can lead to actionable results.

Mobility determinations are identified changes that indicate thebeginning of a trend/pattern for which predictive data insights can thenbe generated and presented as early warning indications pertaining touser activity. The idea is that early warning indications, as well ascontextual understanding of the same, enable users to change theirbehavior to affect a predicted outcome before it is too late to so.Mobility determinations provide a deeper level of understanding of userinteraction that are reflective of how deeply a student is currentlyengaging with content of an organization (e.g., a class) within a recentperiod of time relative to user interactions by other users (e.g.,classmates and/or teachers). For instance, current user activity of auser (e.g., within a recent predetermined period of time) iscomparatively evaluated in a number of different ways, including:directly with historical user activity of that user; and relative touser activity (current and/or historic) of other users (e.g.,classmates, teachers). Notably, as a group of students is beingevaluated collectively (e.g., user activity of a student relative to itsother classmates both current and past), subtle changes in userinteractions/behaviors over a short period of time, that are notindicative of a trend/pattern, can be ignored. In some instances, thoseuser interactions and behaviors can still become a trend/pattern, forexample, when a subsequent round of evaluation commences. Appliedbusiness logic rules identify thresholds for evaluating changes instudent behavior/interactions relative to current and past performanceof a class collectively. For example, a business logic rule is used toset threshold that defines how large a change (deviation) in useractivity is required to even register as a mobility determination.

While a relative evaluation between classmates provides an effectivebaseline to evaluate student behavior, it is also to be recognized thatstudent behavior can change relative to activity of a teacher/professor.As such, the present disclosure applies trained AI processing thatcorrelates user activity data across multiple relative considerations toautomatically generate accurate and precise predictive insights. Thatis, trained AI processing is applied to correlate user activity data ina relative manner to identify changes in user behavior for a recentperiod of time and provide rationale as to why those changes areoccurring.

Furthermore, mobility determinations and predictive insights of thepresent disclosure are intelligent because processing described hereinfactors in business logic that is reflects real-world considerationswhen identified changes in user behavior. For instance, programmed rulescan be applied that tailor a contextual analysis for to unbiasedlyevaluate behavior of users relative to one another. In an educationalscenario, external factors that may affect user behavior which should becontrolled to remove bias include but are not limited to: life events,school schedules, holidays, device access, pandemics, etc. Throughtraining of a classifier, an AI model can further apply business logicto correlate patterns of user behavior with such external factors andremove potential biases found when correlating raw signal data. Forinstance, relevance analysis is applied that curates/prunes datacorrelations to derive the most contextually relevant mobilitydeterminations. Based on results of relevance analysis, predictive datainsights are automatically generated. Predictive data insights areuniquely distinct from predictive outcomes that exclusively focus onuser task completion and grading of how well a user completed a giventask (e.g., assignment, exam).

As an example, say that signal data pertaining to user engagement withweekly lesson content indicates that a student is at the top of theclass with respect to lesson participation. When a predictive datainsight of the present disclosure is generated, a prediction may come inindicating that the student is on track to have a much lower engagementin lesson participation over the next week (e.g., set as a currenttemporal filter). Notification of such context provides an early warningalert to the student and/or teacher so that the student can adjust itsengagement in lesson participation before it may affect the student'sgrade. Additionally, correlation of user activity data over the currenttemporal filter is used to identify a rationale as to why a lowerengagement level is being predicted. For instance, it may be determinedthat a user damaged its laptop, which is normally used to login, and haslogged in over the last two sessions from another device that is sharedwith other users. This can result in limiting the screen time a studenthas, where historically they may have been more proactive aboutaccessing lesson materials. Additionally, as trained AI processing isapplied to automatically generate mobility determinations and generatepredictive insights therefrom, the correlated data that is used toderive a mobility determination is identifiable. As such, key data thatwas identified as a basis for generating a prediction can be identifiedas talking points to help other users (e.g., a teacher, schooladministrators, parents) understand the context surrounding aprediction. This helps foster actionable intervention to avoid anegative outcome.

Exemplary technical advantages provided by processing described in thepresent disclosure comprise but are not limited to: improved server-sideprocessing for management of user activity data to generate meaningfulearly warning indications that are contextually relevant andunderstandable; generation and application of novel trained AIprocessing that is adapted to generate predictive insights fromcontextual relevance analysis of user activity data of a distributedsoftware platform (e.g., educational software platform); novel relevanceanalysis applied by trained AI processing that analyzes historical andcurrent iterations of user activity data in a contextually relativemanner (e.g., across peers and/or based on user interactions fromdifferent types of users); automatic generation of mobilitydeterminations that are usable to generate predictive insights;automatic generation of predictive insights that are derived fromanalysis of user activity data and are usable as early warningnotifications; an improved user interface (GUI) adapted to providenotifications of predictive insights which are extensible to beintegrated within GUIs of a plurality of applications/servicesassociated with a distributed software platform; improved processingefficiency (e.g., reduction in processing cycles, savingresources/bandwidth) for computing devices when generating and renderingnotifications including predictive insights; reduction in latencythrough efficient processing operations that improve correlation ofcontent among different applications/services including integration ofexemplary notifications inline within different host application/serviceendpoints; improve accuracy and precision in application of trained AIprocessing when generating predictive outcomes associated with a domain(e.g., an education-specific domain); and improving usability of hostapplications/services for users via integration of processing describedherein, among other technical advantages.

FIG. 1A illustrates an exemplary system diagram 100 of componentsinterfacing to enable automatic generation of mobility determinationsfrom which predictive data insights can be generated pertaining to usageof system/service, with which aspects of the present disclosure may bepracticed. As an example, components illustrated in system diagram 100may be executed by an exemplary computing system 401 (or multiplecomputing systems) as described in the description of FIG. 4 . Systemdiagram 100 describes components that may be utilized to executeprocessing operations described in process flow 120 (FIG. 1B), processflow 150 (FIG. 1C) method 200 (FIG. 2 ) as well as processing describedin and associated with visual diagrams of FIGS. 3A-3C and theaccompanying description. Moreover, interactions between components ofsystem diagram 100 may be altered without departing from the spirit ofthe present disclosure. Exemplary components, described in systemdiagram 100, may be hardware and/or software components, which areprogrammed to execute processing operations described herein. In someexamples, components of system diagram 100 may each be one or morecomputing devices associated with execution of a specific service.Exemplary services may be managed by a software data platform (e.g.,educational software platform) that also provides, to a component,access to and knowledge of other components that are associated withapplications/services. In one instance, processing operations describedin system diagram 100 may be implemented by one or more componentsconnected over a distributed network, where a user account may beworking with a specific profile established through a distributedsoftware platform. System diagram 100 comprises user computing devices102; an application/service component 104; a mobility determinationmanagement component 106; a component for implementation of trained AIprocessing 108; and knowledge repositories 110.

System diagram 100 comprises user computing device(s) 102. An example ofa user computing device 102 is a computing system (or computing systems)as described in the description of FIG. 4 . User computing device(s) 102are intended to cover examples where a computing device is a clientcomputing device that is executing an application or service configuredto enable access a distributed software platform.

The application/service component 104 is one or more computer components(hardware, software or a combination thereof) configured to manage hostapplications/services and associated endpoints. As previouslyreferences, the application/service endpoint component 108 interfaceswith other computer components of system diagram 100 to enablemanagement of presentation of exemplary notifications in a contextuallyrelevant manner (e.g., inline with content of a specific hostapplication/service endpoint). An application/service component 104further manages presentation of a GUI usable to present an exemplarynotification and foster user interaction therewith. A hostapplication/service configured to enable execution of tasks by one ormore user accounts. Non-limiting examples of host applications/servicesthat are applicable in the present disclosure comprise but are notlimited to: educational information management applications/services;open-source collaborative framework applications/services (e.g.,MICROSOFT® FLUID® video discussion applications/services; wordprocessing applications/services; spreadsheet applications/services;notes/notetaking applications/services; authoring applications/services;digital presentation applications/services; presentation broadcastingapplications/services; search engine applications/services; emailapplications/services; messaging applications/services; web browsingapplications/services; collaborative communicationapplications/services; digital assistant applications/services; webpagebuilding applications/service; directory applications/services; mappingservices; calendaring services; electronic payment services; digitaldata storage or distributed data storage applications/services; webconferencing applications/services; call communicationapplications/services; language understanding applications/services; botframework applications/services; networking applications/services;social networking applications/services; and suites ofapplications/services that may collectively comprise a combination ofone or more of the previously mentioned applications/services includingeducation-specific versions of any of the previously mentioned hostapplications/services, among other examples. The application/servicecomponent 104 further manages respective endpoints associated withindividual host applications/services, which have been referenced in theforegoing description. In some examples, an exemplary hostapplication/service may be a component of a distributed softwareplatform providing a suite of host applications/services and associatedendpoints. A distributed software platform is configured to providingaccess to a plurality of applications/services, thereby enablingcross-application/service usage to enhance functionality of a specificapplication/service at run-time. For instance, a distributed softwareplatform enables interfacing between a host service related tomanagement of a distributed collaborative canvas and/or individualcomponents associated therewith and other host application/serviceendpoints (e.g., configured for execution of specific tasks).Distributed software platforms may further manage tenantconfigurations/user accounts to manage access to features,applications/services, etc. as well access to distributed data storage(including user-specific distributed data storage). Moreover, specifichost application/services (including those of a distributed softwareplatform) may be configured to interface with other non-proprietaryapplication/services (e.g., third-party applications/services) to extendfunctionality including data transformation and associatedimplementation.

Signal data associated with specific applications/service may becollectively analyzed to generation determinations described hereinincluding those where the mobility determination management component106 and/or component implementing the trained AI processing 108 areexecuting importance/relevance scoring/ranking to automatically generatedeterminations described herein. For instance, application of trained AImodel (or models) may be trained to evaluate not only user activity databut other types of contextual data including past and/or current useractions, user preferences, application/service log data, etc., that areeach associated with one or more user accounts. This additional signaldata analysis may help yield determinations as to how (and when) topresent exemplary GUI notifications to users. Notably, different users(e.g., students, teachers, school administrators, parents) may beutilizing an educational software platform, where GUI notifications canbe contextually rendered from analysis of signal data that is customizedfor a user. Non-limiting examples of signal data that may be collectedand analyzed comprises but is not limited to: device-specific signaldata collected from operation of one or more user computing devices;user-specific signal data collected from specific tenants/user-accountswith respect to access to any of: devices, login to a distributedsoftware platform, applications/services, etc.; application-specificdata collected from usage of applications/services and associatedendpoints; or a combination thereof. Analysis of such types of signaldata in an aggregate manner may be useful in helping generatecontextually relevant determinations, data insights, etc. Analysis ofexemplary signal data may comprise identifying correlations andrelationships between the different types of signal data, wheretelemetric analysis may be applied to generate determinations withrespect to a contextual state of user activity with respect to differenthost application/services and associated endpoints.

The application/service component 104 is further configured to present,through interfacing with other computer components of system diagram101, an adapted GUI that provides user notifications, GUI menus, GUIelements, etc., to manage rendering of message notifications andautomatic notifications thereof.

The mobility determination management component 106 is one or morecomputer components (hardware, software or a combination thereof)configured to execute and manage processing operations related togeneration and provision of mobility determinations, predictive insightsgenerated therefrom as well as generation of exemplary GUInotifications. The mobility determination management component 106 isconfigured to execute any processing operations described herein,including those described relative to system diagram 100 (FIG. 1A),process flow 120 (FIG. 1B), process flow 150 (FIG. 1C), method 200 (FIG.2 ), and processing associated with visual diagrams of FIGS. 3A-3C andfurther described in the accompanying description. It is further to berecognized that an order of execution of processing operations by themobility determination management component 106 may vary withoutdeparting from the spirit of the present disclosure.

As referenced in the foregoing description, the mobility determinationmanagement component 106 and/or the application/service component 104are configured to interface with a component for implementation oftrained AI processing 108 to aid processing in various contextualscenarios. The component for implementation of trained AI processing isconfigured to manage implementation of one or more trained AI models.Implementation of trained AI modeling including creating, adapting,training, and updating of trained AI processing is known to one skilledin the field of art. Trained AI processing is applicable to aid any typeof determinative or predictive processing including specific processingoperations described about with respect to determinations,classification ranking/scoring and relevance ranking/scoring. Moreover,a component for implementation of a programmed software module and/ortrained AI processing 110 may be applied to aid generation of processingdeterminations of other components of system diagram 100. An exemplarycomponent for implementation trained AI processing 108 may manage AImodeling including the creation, training, application, and updating ofAI modeling. In cases where trained AI processing is applied, generalapplication of trained AI processing including creation, training andupdate thereof is known to one skilled the field of art. Above what istraditionally known, trained AI processing may be adapted to executespecific determinations described herein with reference to any componentof system diagram 100 and processing operations executed thereby. Forinstance, an AI model may be specifically trained and adapted forexecution of processing operations comprising but not limited to:collecting and analyzing user activity data; generating mobilitydeterminations; curating mobility determinations; generating predictiveinsights; generating GUI notifications for predictive insights;executing data correlation and relevance analysis; generating confidencescoring for selective output of predictive insights; and generation ofdata for rendering GUI content and updates, among other examples.Exemplary AI processing may be applicable to aid any type ofdeterminative or predictive processing by any components of systemdiagram 100, via any of: supervised learning; unsupervised learning;semi-supervised learning; or reinforcement learning, among otherexamples. Non-limiting examples of supervised learning that may beapplied comprise but are not limited to: nearest neighbor processing;naive bayes classification processing; decision trees; linearregression; support vector machines (SVM) neural networks (e.g.,convolutional neural network (CNN) or recurrent neural network (RNN));and transformers, among other examples. Non-limiting of unsupervisedlearning that may be applied comprise but are not limited to:application of clustering processing including k-means for clusteringproblems, hierarchical clustering, mixture modeling, etc.; applicationof association rule learning; application of latent variable modeling;anomaly detection; and neural network processing, among other examples.Non-limiting of semi-supervised learning that may be applied comprisebut are not limited to: assumption determination processing; generativemodeling; low-density separation processing and graph-based methodprocessing, among other examples. Non-limiting of reinforcement learningthat may be applied comprise but are not limited to: value-basedprocessing; policy-based processing; and model-based processing, amongother examples. Furthermore, a component for implementation of trainedAI processing 108 may be configured to apply a ranker to generaterelevance scoring to assist with any processing determinations withrespect to any relevance analysis described herein. Non-limitingexamples of relevance scoring, and specific metrics used for relevancescoring have been referenced in the foregoing description and aresubsequently described including the description of method 200 (FIG. 2). Scoring for relevance (or importance) ranking may be based onindividual relevance scoring metrics described herein or an aggregationof said scoring metrics. In some alternative examples where multiplerelevance scoring metrics are utilized, a weighting may be applied thatprioritizes one relevance scoring metric over another depending on thesignal data collected and the specific determination being generated.Results of a relevance analysis may be finalized according to developerspecifications. This may comprise a threshold analysis of results, wherea threshold relevance score may be comparatively evaluated with one ormore relevance scoring metrics generated from application of trained AIprocessing.

Continuing examples where a domain is education, a proprietary deeplearning model (attention model) is built and trained to identifybehavioral patterns that predict individual activity of a student givena sequence of student interactions. As previously indicated, studentinteractions are considered relative to actions by other students (e.g.,classmates). The model applies a proprietary, student-centric(personalized) self-attention mechanism to consider patterns ofclassmates along with the patterns of the student of interest whenmaking a prediction on said student. A deep learning model of thepresent disclosure focuses on evaluation of user activity rather thanperformance outcomes and is therefore able to provide interpretableoutcomes that prescribe actionable suggestions to users (e.g.,educators, parents). Consider an example where a student is less activein recent weeks but not to the level at which the teacher and/ordescriptive statistics indicate a significant drop. This type ofanalysis may be useful to a teacher enabling identification of anytrends (e.g., down-trend) providing an early warning indication to theeducator with enough context to enable proactive intervention.

A trained AI model is generated to be a robust model that can understandintricacies of domain-specific scenarios. For instance, millions of logsof user data are collected, aggregated, analyzed, and used to train anAI model for a contextual understanding of user activity over aneducational year. For training purposes, a rolling window may beimplemented to specify a time period (temporal filter) across aggregateddata. Training tunes an AI model to understand scenarios that may impactstudents as a whole (e.g., over specific time periods) and avoid biasagainst an individual student in an assessment of that student. Forinstance, student participation may be less over certain periods of timeduring a school year because schools may be on break (e.g., winterbreak, spring break, summer break). If students have days off, thatshould not be counted against them as a reason why their participationmay be lower over a given time period. Similarly, real-time execution ofa trained AI model is adapted, through training and implementation ofbusiness logic (applied business logic rules), to provide context forexecuting an unbiased assessment of a student. The business logic rulesare configured for an education domain of the educational softwareplatform and used to evaluate a relevance of the mobility determinationsgenerated by a trained AI model. In some examples, specific businesslogic rules may further be directed to contextual scenarios that arespecific to a school/university, school district, school, class/course,etc. In this way, a trained AI Model can identify students in similarcircumstances as a baseline for evaluating user behavior.

One type of attention model may focus on user interactions pertaining toassignments issued for a class/course. In alternative examples, anexemplary attention model may be trained to focus on other educationalaspects including but not limited to: user participation; attendance;user collaboration; examinations, or a combination thereof. It is to berecognized that in an assignment example, any of those previouslymentioned signals can be evaluated relative to the issuance of anassignment, homework, testing, etc. For each student assignment, allsignals (signal data) that correspond to it are taken as input. Aspreviously indicated, signal data may be signal data pertaining toaccess by a specific user relative to a student assignment; signal datapertaining to access by other users (e.g., classmates, teachers,parents) relative to a student assignment; signal data pertaining tohistorical access by one or more users relative to past (and/or similar)student assignments, among other examples.

Signal data collected for a specific event (e.g., student assignment)are treated as a series of events in time, each of which is a collectionof features. Specific events are fed to the deep learning modeling whichis trained to generate quantiles (or quartiles) for classifyingpredictions. For every training sequence, each event is composed ofmultiple features that are separately encoded numerically to anappropriate vector according to their underlying input variable type.Signal data indicated as events contain multiple features each of whichis broken down to several input variables. Non-limiting examples of suchvariables comprise but are not limited to: user identification;item/entity identification; host application/service endpoint; deviceidentification (e.g., used for access); signal data type; timestamp data(e.g., start time/end time; and type of access (e.g., continuous,intermittent), among other examples.

Feature representations are concatenated to an event representation. Forexample, encoded variables for each feature are concatenated andembedded (or projected) to a lower dimensional, dense vector,representation. The lower dimensional vector representations are furtherconcatenated to provide a single event representation vector. Singleevent representation vectors are generated for each event in a session.Those single event representation vectors are further evaluated byweighting them by their importance using an attention mechanism applyinga linear combination to yield an aggregate session representation.Session representations in semantic session space are used as input to aprediction layer. An attention mask is generated and used to create aweighted sum of event representations, yielding the aggregate sessionrepresentation.

To avoid a scenario where the trained AI modeling exploits unintendeddata to make successful predictions, processing is applied to categorizeand understand the patterns that the trained AI modeling exploits. Modelattention applied in the present disclosure focuses on specific inputsto indicate which parts of the input contributed most to a model'sdecision. The session representation may then be fed to a fullyconnected network layer (that ends with a Softmax) on the number ofquantiles (or quartiles) that are desired for prediction classification(e.g., top twenty-five percent, top fifty percent, bottom fifty percentbottom twenty-five percent). Ordinal regression loss (or the like) isthen applied on the outcome to generate predictions.

Trained AI processing of the present disclosure is further configured toimplement additional trained AI modeling, in parallel with a trainedattention model, for comparison evaluation and improvement of accuracyand precision of exemplary modeling. For instance, a trained AI modelmay be implemented based on decision trees for comparative evaluation.Implementation of AI modeling using decision trees is known to oneskilled in the field of art. Success of trained AI modeling is evaluatedby criteria that comprising: accuracy in target predictions; beatingsimpler, baseline, approaches; identifying meaningful and interpretablebehaviors captured by the model; and robustness to bias. Using eitherknowledge distillation, or training from scratch, a decision tree/randomforest model may be applied to develop a baseline for generatingrelevant predictions. This processing may comprise applyinghuman-defined business logic (e.g., applicable business rules)configured to target features of interest, thereby turning sequences ofevents into structured tabular data. The structured tabular data is thenintroduced to a decision-tree model, which can be further utilized tocurate predictions.

Once a successful model is generated, it is important to make sure thatbias is avoided, for example, based on a specific user account and/orclass/course). In doing so, additional processing may be executedcomprising one or more of: evaluating the impact of masking suchfeatures; retraining AI modeling without such features to quantify thecontribution of personalization to success of the modeling; investigatealgorithmic mitigation approaches (e.g., demographic parity) to set adiscrimination threshold per group; and execute processing that swapsout attributes of identification (e.g., user account and/orclass/course) with random identifiers. Developers can apply one or moreof these approaches to tune a trained AI model based on threshold foraccuracy and/or precision with respect to results of a trained AI model.

When a model is not successful in proving a hypothesis there are manypotential points of failure that can mask each other. Model iteration isapplied in training to attempt to isolate, uncover and mitigatepotential technical errors. Non-limiting examples of such processing maycomprise but is not limited to: verify convergence of the trained AImodel on an easier/trivial hypothesis or mock data; investigate learningprocess using learning telemetry graphs (e.g., rate of change fordifferent layers can indicate there is a gradient propagation issue andlead to the addition of stabilization/regularization layers);hyperparameter tuning and architecture search; introducing newengineered features such as daily/weekly event aggregations to sessiondata and evaluate impact on performance; and evaluate impact of datapooling versus learning a model per class/course to investigate datadrift related issues, among other examples. Developers can apply one ormore of these approaches to tune a trained AI model based on thresholdfor accuracy and/or precision with respect to results of a trained AImodel.

Additionally, when working within a specific domain (e.g., education),there are additional considerations in correctly applying a successfultrained AI model to achieve the desired effect of reporting on mobility.For an education-specific model, attention may be given to remainingassignment duration. This can help identify patterns of user behaviorthat can indicate whether the user is on track for assignment completion(e.g., normally late in starting assignments or otherwise activitysuggests user is on pace to complete assignment on time) or deviatingfrom prior trends. For instance, when a student receives a newassignment, the trained AI model will be applied after every Xevents/periodically and until Y % of the assignment duration remains. Ywill be determined according to the model report (e.g., threshold targetmay be set similar to something like >30 percent).

Another consideration pertains to mobility resolution and baseconditioning. Mobility is a substantial change in quantiles (orquartiles) over time and requires defining the following: resolution(what is substantial). For example, in quantiles (or quartiles) a changecan be detected from top twenty-five percent to middle of the class(e.g., top fifty percent). A student that is on average graded aroundthe top three-fourths of a class will tend to fluctuate between the topquantiles (or quartiles). As such, a prediction that calls that studentin a quantiles (or quartiles) may not be truly informative. There are afew workarounds to avoid this including but not limited to: increasequantiles (or quartiles) resolution in labeling; reporting onsubstantial mobility effect size only (e.g., switching from toptwenty-five percent quartile to bottom twenty-five percent quartile; andmanaging base conditioning (derivation over time) with respect tostudent performance, among other examples. A trained AI model mayimplement one or more of these approaches when a user based as a wholeand/or evaluating specific groups of users. Developers can apply one ormore of these approaches to tune a trained AI model based on thresholdfor accuracy and/or precision with respect to results of a trained AImodel.

As referenced in the foregoing description, knowledge repositories 110may be accessed to manage data aiding operation of any other computercomponents described in system diagram 100. Knowledge resources compriseany data affiliated with a software application platform (e.g.,Microsoft®, Google®, Apple®, IBM®) as well as data that is obtainedthrough interfacing with resources over a network connection includingthird-party applications/services. Knowledge repositories 110 may beresources accessible in a distributed manner via network connection thatmay store data usable to improve processing operations described herein.Examples of data maintained by knowledge repositories 110 comprises butis not limited to: activity data logs; generated mobilitydeterminations, predicted insights and GUI notifications; collectedsignal data (e.g., from usage of an application/service,device-specific, user-specific); telemetry data including past andpresent usage of a specific user and/or group of users; data forexecution of application/services including host application/servicesand associated endpoints; corpuses of annotated data used to build andtrain AI processing classifiers for trained AI modeling; access toentity databases and/or other network graph databases usable forevaluation of signal data; web-based resources including any dataaccessible via network connection including data stored via distributeddata storage; trained bots including those for natural languageunderstanding; software modules and algorithms for contextual evaluationof content and metadata; and application/service data (e.g., data ofapplications/services managed by the application/service component 104)for execution of specific applications/services including electronicdocument metadata, among other examples. In even further examples,telemetry data may be collected, aggregated and correlated (e.g., by aninterfacing application/service) to further provide computer componentsof system diagram 100 with on demand access to telemetry data which canaid determinations generated thereby.

FIG. 1B illustrates an exemplary process flow 120 for generatingmobility determinations and provision of predictive data insights to endusers, with which aspects of the present disclosure may be practiced.Process flow 120 illustrates a non-limiting example pertaining toprocessing of user activity data associated with an educational softwareplatform. Process flow 120 highlights a flow of data relative to trainedAI modeling (e.g., via a component for implementation of trained AIprocessing 108) when providing exemplary GUI notifications comprisingpredictive insights. Process flow 120 comprises: students 122; a firstinteraction 124 of applications/services providing user activity data; adata storage of user activity data 126; the component for implementationof trained AI processing 108 (e.g., to manage training of AI modelingand real-time exposure of trained AI modeling); a second interaction 128of applications/services providing a user experience; and other endusers 130 (e.g., educators, school leaders, other students, parents).

As a starting point, process flow 120 shows students 122 (e.g.,individual students) which are intended to be an example of users of adomain-specific software data platform. As previously indicated,students 122 may use applications/services of an educational softwareplatform, where logs of user access instances to application/services ofthe educational software data platform may be created. Logged data ofuser activity is stored for individual instances of user access and mayfurther be aggregated by user (or as a group of users).

Process flow 120 further illustrates a first interaction 124 ofapplications/services that provide user activity data. The firstinteraction 124 is intended to be a representation of the capture ofuser activity data for analysis. Captured activity data is stored on adata storage of user activity data 126 (e.g., distributed data storage)for subsequent access to execute contextual analysis as describedherein. Importantly, it is recognized that user activity data is stored(and accessed) in compliance with privacy laws and regulations.Furthermore, exemplary modeling is trained and executed in a privacycompliant manner where developers never see data due to compliancerequirements/restrictions on access to user activity data.

Moreover, process flow 120 illustrates an interaction of the componentfor implementation of trained AI processing 108, which is used to managetraining of AI modeling and real-time exposure of trained AI modelingfor generation of predictive insights from exemplary contextualanalysis. Trained AI processing (e.g., one or more trained AI models)may be generated, trained and exposed for real-time (or near real-time)analysis of user activity data.

As a result of application of a trained AI model, a second interaction128 of applications/services provides an end-user experience thatsurfaces, in a GUI of a host application/service endpoint, GUInotifications that comprise predictive insights. In the example shown inprocess flow 120, exemplary GUI notifications, comprising predictiveinsights, are provided to other end users 130 (e.g., educators, schoolleaders, other students, parents). For instance, GUI notifications ofpredictive insights are presented in user-specific renderings ofapplication/services associated with an educational software platform.In alternative examples of process flow 120, GUI notifications can alsobe presented to the students 122 (individual students) for whom thepredictive data insights are generated. This may provide a way in whichusers can monitor their own user activity and interaction withapplications/services of an educational software platform.

FIG. 1C illustrates an exemplary process flow 150 for generatingmobility determinations within an educational software platform, withwhich aspects of the present disclosure may be practiced. Process flow120 highlights processing interactions of trained AI modeling (e.g., viaa component for implementation of trained AI processing 108) whengenerating predictive insights. Process flow 150 comprises: inputprocessing 152; contextual representation 154; prediction processing156; and comparison processing 158. Further, process flow 150 is labeledwith additional numbering (steps 1-6) to help guide one non-limitingexample of flow of data when generating predictive insights.

Following the steps (1-6) of process flow 150, step 1 illustrates theintroduction of input 152 into a system in the form of user activitydata. As indicated in the foregoing description, user activity data maycomprise data pertaining to user interactions with applications/servicesof an educational software platform. For instance, user activity data islogged for all students that are assigned to a specific assignment (andassociated metadata) as well as all assignment-related events that occurfor those students within an educational software platform.

Steps 2 and 3 of process flow 150 illustrates the application of atrained AI model to generate feature representations from contextualanalysis of user activity. From analysis of user activity data by atrained AI model, a contextual representation 154 of the user activitydata is generated. Generation of a contextual representation 154 of useractivity data comprises generation of a student representation (step 2)of each student assigned to an assignment. Student representations are afeature representation of student user activity pertaining to anassignment. The proprietary attention modeling of a trained AI model isthen utilized to generate an event representation (step 3) for eachstudent representation. Event representations are concatenated examplesof student representations. For example, encoded variables for eachfeature are concatenated and embedded (or projected) to a lowerdimensional, dense vector, representation. The lower dimensional vectorrepresentations are further concatenated to provide a single eventrepresentation vector. Single event representation vectors are generatedfor each event in a session. Those single event representation vectorsare further evaluated by weighting them by their importance using anattention mechanism applying a linear combination to yield an aggregatesession representation.

Step 4 of process flow 150 is the generation of a contextualrepresentation of one or more students associated with an assignment. Acontextual representation is aggregated session representation which isan aggregations of single event representations as a session. Sessionrepresentations in semantic session space are used as input to aprediction layer. An attention mask is generated and used to create aweighted sum of event representations, yielding the aggregate sessionrepresentation.

Step 5 of process flow 150 illustrates the application of a predictionlayer 156 that analyzes session representations (generation result ofcontextual representation 154) to generate one or more predictions fromanalysis of user activity data. As indicated in the present disclosure,a trained AI model analyzes data correlations, relative to theapplication of business logic rules, to generate predictions fromcontextual analysis of user activity data.

Step 6 of process flow 150 illustrates the comparison evaluation 158,resulting in the generation of predictive outcomes. Predictive outcomescomparatively evaluate students relative to the user activity of otherstudents. Predictive outcomes are the basis for predictive insights,which can be presented to end users of an educational software platformas early warning indications.

FIG. 2 illustrates exemplary method 200 related to automatic generationof mobility determinations and management of predictive data insightsderived therefrom, with which aspects of the present disclosure may bepracticed. As an example, method 200 may be executed across an exemplarycomputing system 401 (or computing systems) as described in thedescription of FIG. 4 . Exemplary components, described in method 200,may be hardware and/or software components, which are programmed toexecute processing operations described herein. Non-limiting examples ofcomponents for operations of processing operations in method 200 aredescribed in system diagram 100 (FIG. 1A), process flow 120 (FIG. 1B)and process flow 150 (FIG. 1C). Processing operations performed inmethod 200 may correspond to operations executed by a system and/orservice that execute computer modules/programs, software agents,application programming interfaces (APIs), plugins, AI processingincluding application of trained data models, intelligent bots, neuralnetworks, transformers and/or other types of machine-learningprocessing, among other examples. In one non-limiting example,processing operations described in method 200 may be executed by acomputer component such as: a mobility determination managementcomponent 102; an application/service component 104; a component forimplementation of trained AI processing 108, or a combination thereof.In distributed examples, processing operations described in method 200may be implemented by one or more computer components connected over adistributed network. For example, computer components may be executed onone or more network-enabled computing devices, connected over adistributed network, that enable access to user communications.

Method 200 begins at processing operation 202, where user activity dataof a distributed software platform is collected. The user activity datais raw signal data received from a plurality of applications or servicesassociated with the educational software platform. An educationalsoftware platform is a non-limiting example of a distributed softwareplatform. A distributed software platform is a software systemconnecting components thereof over a distributed network connection.Implement of components to enable operation of components of a softwaredata platform over a network connection are known to one skilled in thefield of art. For instance, a distributed software platform may bebacked by one or more services to enable the distributed softwareplatform to be implemented in a variety of technical distributed datascenarios including but not limited to: software as a service (SaaS),platform as a service (PaaS) and infrastructure as a service (IaaS).Moreover, the distributed software platform may support many differentprogramming languages, tools, and frameworks, etc., including bothorganizationally proprietary systems (e.g., MICROSOFT®-specific) andthird-party software and systems including those of independent softwarevendors.

Collection (processing operation 202) of user activity data occurs asusers perform different activities through a software data platform. Forinstance, a user login to an educational software platform may create asession where signal data may be logged relative to a session of useraccess to one or more applications/services of a software data platform(e.g., educational software platform). User activity data is recognizedas application-specific signal data or service-specific signal data thatpertains to user activity received through applications/servicesassociated with the educational platform. Importantly, user activitydata that is of interest to the present disclosure is activity dataassociated with parameters that are behavioral and can be changedrelative to a users' interaction with applications/services of theeducational platform. Steering away from parameters that are specific tocertain classes courses, teaching methods, etc., helps focus an analysison how a user interaction with specific components,applications/services, etc., can be improved while removing potentialfor bias. For instance, a user being notified to start an assignmentearlier in time can help change a behavior of the student andpotentially avoid missing an assignment deadline. Non-limiting examplesof user activity data with an educational platform comprises but are notlimited to user activity data pertaining to: login information; access(including time and amount of access) to specific content, assignments,posts, feedback/comments, resources, meetings, tests/exams, etc.;starting and completion timing for completing tasks, assignments, etc.;collaborative interactions between users and/or teachers and students;modification of content; posting of content including assignments,exams, etc.; use of features, emojis, etc. and grading, among otherexamples. Other examples of signal data pertaining to user interactionswith an educational platform are known to one skilled in the field ofart.

At processing operation 204, logs of user activity data are stored forrecall. For instance, user activity data is stored on one or moredistributed data storages (or distribute data storage systems). In oneexample, user activity data is stored via data storage of a file hostingservice or a document management storage system. Importantly, it isrecognized that user activity data is stored (and accessed) incompliance with privacy laws and regulations. Furthermore, exemplarymodeling is trained and executed in a privacy compliant manner wheredevelopers never see data due to compliance requirements/restrictions onaccess to user activity data. As identified in the foregoingdescription, signal data may be logged relative to a session of useraccess to one or more applications/services of a software data platform(e.g., educational software platform). For a class of students there islikely a plurality of logs each day of activity. In an educationalspace, a predictive outcome can either be straight forward (e.g., will astudent turn in an assignment on time) or abstract (e.g., is a usergoing to reduce engagement over a given time period). For instance, asystem of the present disclosure can be used to predict what will happenin a given week and provide those predictions as predictive insightsand/or early warning indications of decaying performance of a studentbefore it is too late for the teacher to take action and positivelyinfluence a behavior of the student.

Flow of method 200 then proceeds to processing operation 206. Atprocessing operation 206, trained AI modeling is generated that isadapted to contextually analyze the user activity data (e.g., of theeducational software platform). Generation and management of a trainedAI model including training of one or more classifiers is known to oneskilled in the field of art. Above what is traditionally known,processing is executed to feed the trained AI model with raw signalspertaining to user activity data of an educational software platform. Anexemplary AI model is adapted, through training of a corpus of relevanttraining data (including sample user activity data and business logicrules), to find patterns on its own based on analysis of: activity datapertaining to current user interactions of one or more users within acurrent temporal filter; historical user activity data pertaining tointeractions of a specific user (or group of users) with specificcomponents, applications/services, users, etc., of the educationalplatform; historical user activity data identifying how a peer of a user(e.g., classmates) interact with specific components,applications/services, users, etc., of the educational platform; andhistorical user activity data identifying how a teacher (e.g., of thestudent/classmates) interact with specific components,applications/services, users, etc., of the educational platform, amongother examples.

Furthermore, generation (processing operation 206) of the trained AImodel comprises building a proprietary attention model that is tailoredto work with contextual data of an educational software platform.Processing for generating a propriety attention model has been describedin the foregoing description including the description of system diagram100 (FIG. 1A). As previously described, an exemplary trained AI modelmay be adapted to generate a domain-specific attention model to evaluateuser activity data associated with users of a specific domain. In theexample where a domain is education, a student-centric (personalized)self-attention model is generated that is adapted to consider patternsof activity of a student of interest relative to patterns of activity ofother users (e.g., classmates and/or teachers/professors) when making aprediction on said student of interest. A deep learning model of thepresent disclosure focuses on evaluation of user activity rather thanperformance outcomes and is therefore able to provide interpretableoutcomes that prescribe actionable suggestions to users (e.g.,educators, parents).

As an example, contextual analysis to build an attention model focuseson generating mobility determinations derived from analysis of: useractivity (individually and comparative with other users) includinginteractions of students; activity of a student relative to otherstudents; and activity of a student responsive to interactions by/with ateacher (or teachers), among other examples. Through iterative training,a trained AI model is configured to weight these interactions todetermine most relevant patterns.

Furthermore, training of an AI model further comprises deriving and/orapplying business logic rules that relevant to a specific domain (e.g.,education). In an educational example, application of business logicrules help tailor an attention model to identify students in similarcircumstances as a baseline for evaluating user behavior in a relativemanner. Exemplary business logic rules are configured for an educationdomain of the educational software platform and used to evaluate arelevance of the mobility determinations generated by a trained AImodel. In some examples, specific business logic rules may further bedirected to contextual scenarios that are specific to aschool/university, school district, school, class/course, etc. In thisway, a trained AI Model can identify students in similar circumstancesas a baseline for evaluating user behavior.

Exemplary business logic rules may comprise rules that are specific tobusiness decisions and rules that are data driven. Examples of businesslogic rules that are specific to business decisions comprise but are notlimited to rules that identify: what types of data to analyze (e.g.,user actions pertaining to assignments); duration of analysis (e.g., fora given data type such as student assignment); how to handle data overgiven time periods (e.g., relative to a school calendar); rules that arespecific to evaluation of users across a specific school/university,school district, school, class/course, etc.; attributes of user activitydata to prioritize or to avoid (to mitigate bias); and how to executecomparison of users (e.g., student versus classmates), among otherexamples. Business logic rules that are specific to business decisionsare preset by developers and applicable regardless of the correlationsgenerated as a result of data analysis. As previously identified,business logic rules further comprise application of rules that are datadriven generated as a result of training and analysis of domain-specificdata. For instance, analysis of user activity data in training mayidentify data correlations that are most impactful to generatingmobility determinations and/or predictive insights. Examples of businesslogic rules that are data driven comprise but are not limited to rulesthat identify: define mobility (e.g., how to evaluatequantiles/quartiles during data analysis); placing a weighting (e.g.,prioritizing or de-prioritizing) certain data correlations whengenerating mobility determinations and/or predictive data insights;thresholds for identifying data indicating a trend/pattern versus anoutlier (e.g., need X number of instances in data analysis to generate apredictive insight using the data); setting discrimination thresholds(e.g., per user group); identifying when to correlate data of certainusers (e.g., teachers, classmates, parents) with a specific student;what data correlations to identify as talking points (e.g., based onrelevance analysis of data correlations to a mobility determination);and how to use data to generate predictive insights and GUInotifications (e.g., what data is most relevant to be included in afirst-level representation of a data insights and what to include asnested data (e.g., a second-level representation)), among otherexamples.

Training processing utilizes applied and/or derived business logic rulesto curate and prune correlations from raw user activity data. This maycomprise analysis as to how data is distributed and feed differentthresholds into the business layer to determine importance of mobilitydeterminations. As indicated, data thresholds may be set by developersbased using business logic rules, where threshold can be used toidentify trends/patterns in data as well as filter out signals that aretoo sparse. In one example, confidence scoring (relevance scoring) isgenerated and applied to help derive data correlations that are mostimportant to establish mobility determinations. Furthermore, oncemobility determinations are identified, confidence scoring (relevancescoring) may be applied to help curate mobility predictions. Forinstance, not all mobility predictions are created equal. Confidencescoring (relevance scoring) is applied, relative to business logicrules, to determine which mobility determinations are most relevant to aspecific aspect of user behavior that is being analyzed (e.g., anevaluation of user interactions pertaining to an assignment). In oneexample, a threshold pertaining to relevance scoring is set to identifymobility determinations (that satisfy the threshold) for output duringreal-time analysis. This can help identify most impact mobilitydeterminations and improve processing efficiency when generatingpredictive data insights as well as identify talking points as the keydata correlations that resulted in generated mobility determinations.

Train of AI processing may further comprise generating an AI model thatis tuned to reflect specific metrics for accuracy, precision and/orrecall before a trained AI model is exposed for real-time (nearreal-time) usage. Developers may set thresholds for specific metrics tomake sure that a trained AI model is operating as expected. Thresholdsfor metric evaluation of a specific trained AI model may vary, dependingon developer specifications, without departing from the spirit of thepresent disclosure.

Once a threshold (or thresholds) is met for exposing a trained AI model,flow of method 200 proceeds to processing operation 208. At processingoperation 208, the trained AI modeling is exposed for real-time (or nearreal-time) evaluation of user activity data.

At processing operation 210, trained AI modeling (e.g., a trained AImodel) is then applied to generate predictive insights from contextualrelevance analysis of the user activity data. In doing so, a currenttemporal filter (e.g., user activity data over the last week) is set andused to frame analysis of the user activity data of one or more usersrelative to current user activity data of other users and/or historicaluser activity data of the user and other users. A current temporalfilter is a time period (parameter) used to evaluate recent useractivity data. For instance, a current temporal filter may be applied toevaluate one or more users over a recent period of time (e.g., a day,week, month). It is to be recognized that developers can set a currenttemporal filter to any specific time period without departing from thespirit of the present disclosure.

As indicated in the foregoing description, a trained AI model is adaptedto execute a plurality of processing operations to generate predictiveinsights. For instance, the trained AI modeling generates (processingoperation 212) a plurality of mobility determinations that identifychanges in patterns of user behavior over a current temporal filterassociated with the user activity data. A trained AI model relies on thetraining of an associated classifier (or classifiers) to analyzegenerated representations of data and derive mobility determinationstherefrom. Mobility determinations are identified changes that indicatethe beginning of a trend/pattern for which predictive data insights canthen be generated and presented as early warning indications pertainingto user activity. The idea is that early warning indications, as well ascontextual understanding of the same, enable users to change theirbehavior to affect a predicted outcome before it is too late to so.Mobility determinations provide a deeper level of understanding of userinteraction that are reflective of how deeply a student is currentlyengaging with content of an organization (e.g., a class) within a recentperiod of time relative to user interactions by other users (e.g.,classmates and/or teachers).

In correlation with applied business logic rules, mobilitydeterminations are aimed at identifying changes in quantiles (quartiles)of user activity that are substantial and potentially indicative of atrend in a negative (or positive) direction. Notably, as a group ofstudents is being evaluated collectively (e.g., user activity of astudent relative to its other classmates both current and past), subtlechanges in user interactions/behaviors over a short period of time, thatare not indicative of a trend/pattern, can be ignored. In someinstances, those user interactions and behaviors can still become atrend/pattern, for example, when a subsequent round of evaluationcommences. Applied business logic rules identify thresholds forevaluating changes in student behavior/interactions relative to currentand past performance of a class collectively. For example, a businesslogic rule is used to set threshold that defines how large a change(deviation) in user activity is required to even register as a mobilitydetermination. For instance, a threshold can be set that says a user hasto drop (or increase) from a quantile/quartile pertaining to anevaluation (e.g., assignment activity/interactions) to even register asa mobility determination. In alternative examples, business logic rulescan set thresholds to any value (e.g., a certain percentage) to triggeridentification of a mobility determinations. In further examples, thesame type of thresholds can be used to curate mobility determinations.One system example of the present disclosure is configured to generate aplurality of mobility of determinations (without considering athreshold), where a threshold evaluation is then applied to curate themobility determinations to identify those that are most relevant (e.g.,most substantial relative to the threshold set by developers).

As indicated in the foregoing, generation (processing operation 212) ofmobility determinations evaluates current user activity of a user (e.g.,activity within a current temporal filter)) comparatively in a number ofdifferent ways, including: directly with historical user activity ofthat user; and relative to user activity (current and/or historic) ofother users (e.g., classmates, teachers). While a relative evaluationbetween classmates provides an effective baseline to evaluate studentbehavior, it is also to be recognized that student behavior can changerelative to activity of a teacher/professor. As such, the presentdisclosure applies trained AI processing that correlates user activitydata across multiple relative considerations to automatically generateaccurate and precise predictive insights. As such the plurality ofmobility determinations are generated (processing operation 212) basedon a collective relevance analysis that correlates: data pertaining tocurrent user interactions of a first user that are identified within thecurrent temporal filter; data pertaining to historical user interactionsof the first user that are identified within a historical temporalfilter associated with the user activity data; data pertaining tocurrent user interactions of one or more other users that are identifiedwithin the current temporal filter; and data pertaining to historicaluser interactions of the one or more other users that are identifiedwithin the historical temporal filter.

Similar to a current temporal filter, a historical temporal filter is atime period (parameter) used to evaluate historical user activity data.For instance, a historical temporal filter is framed relative to acurrent temporal filter, where the historical temporal filter is anyprevious user activity that occurred prior to the time period set forthe current temporal filter. In some examples, a specific window ofhistorical user activity data (e.g., last week, last six months, lastyear) is evaluated relative to a current temporal filter. In otherexamples, an entirety of historical user activity data for one or moreusers is evaluated when generating mobility determinations. In at leastone example, business logic rules are applied to determine a relevanttemporal period for one or more filters such as a historical temporalfilter. For instance, an amount of user activity may be a threshold thatis evaluated to help frame a historical temporal filter. If an amount ofuser activity is not satisfied, then certain historical user activitymay not need to be considered to generate a relevant mobilitydetermination.

Further processing operations executed by the trained AI modelingcomprises curating (processing operation 214) the plurality of mobilitydeterminations to generate a curated listing of mobility determinations.The curated listing of the mobility determinations comprises one or morecurated mobility determinations identified based on a thresholdevaluation of the relevance scoring for each of the plurality ofmobility determinations. As previously indicated, a curated listing ofmobility determinations is derived based on an application of businesslogic rules, configured for an education domain of the educationalsoftware platform. Said business logic rules are used to evaluate arelevance of the mobility determinations. In at least one example, anapplication of the business logic rules comprises applying businesslogic rules that assign a weighting to specific types of user activityidentified within the user activity data. For instance, the specifictypes of user activity data are instance of user interactions relativeto the education-specific (educational domain), which may be identifiedas most relevant for correlating data based on results of training of anAI classifier. In one example, relevance scoring is generated for eachof the plurality of mobility determinations relative to the weightingassigned to specific data correlations. When it comes time to providerationale supporting predictive insights (e.g., talking points), theweighted data correlations can be identified as talking points toprovide a user with a rationale as to why a predictive insight wasgenerated (and selected for presentation).

In further examples, relevance analysis for generating a curated listingof mobility determinations comprises application of business logic rulespertaining to thresholds in activity changes. This may occur in additionto (or in lieu) of relevance analysis that generates relevance scoringbased on weighting of data correlations. As previously identified,mobility determinations are aimed at identifying changes in quantiles(quartiles) of user activity that are substantial and potentiallyindicative of a trend in a negative (or positive) direction. A businesslogic rule is used to set a threshold that defines how large a change(deviation) in user activity is required to be included in a curatedlisting of mobility determinations. For example, a threshold can be setthat says a user has to drop (or increase) from a quantile/quartilepertaining to an evaluation (e.g., assignment activity/interactions) tomake the curated listing of mobility determinations. In alternativeexamples, business logic rules can set thresholds to any value (e.g., acertain percentage) to be added to a curated listing of mobilitydeterminations. In at least one example, the curated listing of mobilitydetermines is prioritized (ordered) based on results of this relevanceanalysis. A prioritized (or ordered) listing of mobility determinationsmay be utilized to determine how many mobility determinations to includefor predictive data insight generation.

Flow of method 200 then proceeds to processing operation 216. Atprocessing operation 216, one or more predictive insights may begenerated based on an analysis of the curated listing of the pluralityof mobility determinations. As previously referenced, predictiveinsights may be generated based on evaluation of the prioritized (orordered) listing of mobility determinations derived based on a relevanceanalysis. For instance, N number of predictive insights (e.g., highestpriority/order) may be generated from a curated listing of mobilitydeterminations. In further examples, predictive data insights may begenerated for each of the mobility determinations on the curated listingof mobility determinations.

In additional examples, predictive insights may be generated based onmultiple aggregation of multiple mobility determinations. For instance,say a predictive insight is generated that says a student may miss adeadline for an assignment based on a recent evaluation of user activityvia the collective analysis described in the present disclosure. Thismay be derived based on analysis of user activity of not only the userrelative to past activity of that user on similar assignments (e.g., afirst mobility determination) but also based on analysis of groupactivity between the user and his classmates on similar assignments(e.g., a second mobility determination). In that example, there are twodata correlations provide rationale for a predictive data insight. Assuch, there may be a higher likelihood of confidence in a predictiveinsight based on the number of relevant mobility determinations thatprovide the rationale/basis supporting the predictive insight. Followingthat example, generation of the predictive insights comprises:assigning, by application of the trained AI model, a confidence scoringto the predictive insights based on a correlation, for each of the oneor more predictive insights, with one or more mobility determinationsincluded in the curated listing of the plurality of mobilitydeterminations. In such technical instances, the one or more predictiveinsights are then generated based on a threshold evaluation of theconfidence scoring for each of the one or more predictive insights.Thresholds for evaluating confidence scoring for generation ofpredictive insights may vary according to developer specificationswithout departing from the spirit of the present disclosure.

In additional examples, generation of the one or more predictiveinsights comprises: identifying, from the collective relevance analysisexecuted by the trained AI model, one or more data correlations thatprovide rationale supporting predictive insights (e.g., talking points).Talking points as described herein are support/rationale for apredictive insight that provide insight into why a predictive insightwas generated. Talking points are intended to help a user (e.g.,educator or parent) frame and discuss a trend/pattern of user activity.For example, if a user interaction and proactivity is much lower than ithas traditionally been, it may be helpful to use talking points toidentify the reason behind this trend. As described in previousexamples, contextual analysis of user activity data may yield adetermination that a student is not signing in with their normalcomputing device and instead using a shared computing device of ahousehold, which may be contributing to a lack of user activity.Weighted data correlations can be identified as talking points toprovide a user with a rationale as to why a predictive insight wasgenerated (and selected for presentation). As indicated in theforegoing, talking points are includable in predictive data insights andGUI notifications thereof. In one example, a GUI notification of apredictive insight may provide a first-level representation of apredictive data insight, where additional layers of the GUI notificationmay reveal additional context regarding a predictive insight (e.g.,talking points). A non-limiting example of rendering of talking points,in a layered representation is shown in the progression from FIGS.3A-3C. Talking points may be stored for recall either within a generatedpredictive insight or as an additional data object (or metadata) thatcan be added to a GUI notification of a predictive insight.

Once predictive insights (and associated data objects) are generated,flow of method 200 proceeds to processing operation 218. At processingoperation 218, data pertaining to notification(s) of predictive insightsis managed. For instance, processing operation 218 may comprisegenerating a GUI notification that comprises the one or more predictiveinsights. This may occur through automated processing by trained AImodeling or via an application/service associated with the educationalsoftware platform. In one example, formatting of notifications ispredetermined to populate data fields based on the type of notification(e.g., callout, GUI menu, graph, dynamic timeline) being generated. Inother examples, a format and layout of a notification may be dynamicallygenerated based on analysis of the type of mobility determinations andpredictive insights generated. For example, key data points ofcorrelated data may be expressible in various ways, which may be bestrepresented by evaluating the type of correlated data points torepresent relative to processing capabilities of applications/servicesof the educational software platform. In some technical instances, userinteractions with GUI notifications of predictive data insights maydictate how and when to present data associated with a predictive datainsight. For example, a user action may select a GUI feature of asurfaced GUI notification to reveal additional context (e.g., talkingpoints) regarding predictive data insights. In further examples, GUInotifications are configured to enable users to take immediate action toaddress findings of a predictive insight. GUI notifications may compriselinks to automatically initiate message notifications, emailnotifications, and links to additional telemetric data (e.g., graphs,charts, statistics), among other examples.

In some examples, predictive insights and GUI notifications aredynamically generated during real-time (or near real-time) useroperation of an educational software platform. In other examples,predictive insights and GUI notifications thereof are stored for recalland presented to a user during subsequent access to the educationalsoftware platform. For instance, processing operation 218 may comprisestoring, on a distributed data storage that is associated with theeducational software platform, the predictive insights and associatedGUI notifications. As previously referenced, a data storage of a filehosting service or a document management storage system is used to storeand manage data for a trained AI model, generated mobilitydeterminations, generated predictive insights (and associated dataobjects) and generated GUI notifications comprising predictive insights.

Whether or not predictive insights are generated and presented inreal-time (or near real-time), data for rendering the one or morepredictive insights is transmitted (processing operation 220) to anapplication or service of the educational software platform. Ininstances where a GUI notification has already been generated, the datafor rendering the one or more predictive insights is data for renderingthe GUI notification comprising the one or more predictive insights.

In some examples, rendering of a GUI notification may be processingexecuted by an exemplary computing device (e.g., computing system 401 ofFIG. 4 ). Flow of method 200 may then proceed to processing operation222. At processing operation 22, the one or more predictive insights arerendered in the GUI notification that is displayable through a GUI ofthe application or service of the educational software platform. Inline,as referenced in the present disclosure, is intended to refer totechnical instances where data is embedded as a content portion (dataobject), among other content portions, that is displayable within arendering of a host application/service. For instance, a GUInotification and associated data are embedded components that is appearsas content portions within a GUI of a host application/service endpoint.An application or service of the educational software platform uses thedata for rendering the GUI notification to present a predictive insight.Non-limiting examples of GUI notifications are provided in theprocessing device views illustrated in FIGS. 3A-3C.

Furthermore, in examples where a GUI notification is rendered in GUI ofan application/service, flow of method 200 may proceed to processingoperation 224. At processing 224, presentation of the GUI notificationis updated based on user interaction (user action) with the GUInotification. For instance, a user may select a GUI element requestingadditional contextual information about a predictive insight. In otherinstances, a user (e.g., teacher, administrator, parent, student) maywish to follow-up with another user with respect to a prediction. Infurther instances, users may provide user feedback regarding theaccuracy and relevance of a predictive insight and/or GUI notification.

User feedback may be used to continuously update a trained AI model toimprove predictions and generate the most contextually relevantinsights. As such, any user activity, including user feedback receivedwith respect to GUI notifications and/or predictive insights may be usedto update (processing operation 226) the AI modeling. For instance,training data and/or a corpus of additional user activity data may beadded and further used to build off of previous iterations of thetrained AI modeling. Method 200 may then return to processing operation202 to collect additional user activity data for subsequent analysisthereof. As previously indicated, predictive insights may becontinuously generated (e.g., using a new current temporal filter) toframe user activity as a student progresses throughout a class/course.Subsequent predictive insights may be relative to the same assignment inwhich previous predictive insights were generated as well as newassignments (or evaluation of other aspects of user activity).

FIGS. 3A-3C illustrate exemplary processing device views associated withuser interface examples for an improved user interface that is adaptedfor generation and management of predictive data insights frommanagement of mobility determinations, with which aspects of the presentdisclosure may be practiced. FIGS. 3A-3C provide non-limiting front-endexamples of processing described in the foregoing including systemdiagram 100, process flow 120 (FIG. 1B), process flow 150 (FIG. 1C), andmethod 200 (FIG. 2 ).

FIG. 3A presents processing device view 300, illustrating a GUI of acollaborative communication application/service (e.g., MICROSOFT®TEAMS®) that is configured as part of an educational software platform.Processing device view 300 illustrates the presentation of a pluralityof exemplary GUI notifications inline with other content of thecollaborative communication application/service. Inline, as referencedin the present disclosure, is intended to refer to technical instanceswhere data is embedded as a content portion (data object), among othercontent portions, that is displayable within a rendering of a hostapplication/service. Processing operations for generation of exemplarynotifications that comprise predictive insights have been described inthe foregoing description.

In the example shown in processing device view 300, a digital engagementGUI menu 302 presents multiple GUI notifications (respectively first GUInotification 304 and second GUI notification 306) pertaining toevaluation of user engagement with digital content that is associatedwith an educational software platform. GUI notifications 304 and 306comprise forms of both descriptive data insights and predictive datainsights. From contextual analysis of user activity data (e.g., by thetrained AI modeling) a predictive insight 312 is generated providing apredictive talking point (rationale) as to why student activitydecreased over the last week. For example, predictive insight 312identifies a rationale for the reduced amount of student activity thatis derived from analysis of mobility determinations and key points ofcorrelated data that support the mobility determinations. As can be seenin processing device view 300, predictive insight 312 suggests that homedevice sharing issues (e.g., amongst students) may be the result forless student activity through the educational software platform. Forinstance, it may be determined correlation of user activity data mayprove that a user is not logging in from its assigned laptop and isinstead logging in from a tablet that is shared with multiple otherusers. This provides an understandable explanation for a teacher tofollow-up on and help remedy the situation.

In the example shown in GUI notification 306, a predictive insight 314is generated to aid descriptive insights identifying analytics of usercontent postings relative to an educational software platform. As can beseen in processing device view 300, predictive insight 314 suggests thatteacher feedback provided to students for a last posting was theunderlying trigger for increased user posting activity. This predictiveinsight was likely derived from the correlation processing of useractivity data for students of a class relative to user activity data bythe teacher of the class. This provides an understandable rationale fora teacher to understand a direct correlation between its activity in theeducational software platform and an effect on its students.

GUI notification 308 provides another example of a predictive insight.Similar to the previously described GUI notifications, GUI notification308 comprises both descriptive insights and predictive insights. In theexample shown, a descriptive insight (“Shay Daniel hasn't started Comedyof Errors assignment yet”) is provided. Based on a comparative analysisof user activity data for a current temporal period (e.g., current week)of a specific user relative to other users (e.g., classmates) andfurther relative to historical activity data of the user and other usersof a class, predictive insight 316 is generated providing a predictedoutcome that anticipates that a student will miss the assignmentdeadline. Again, this level of prediction is considered an early warningindication because the assignment due date has not passed and there isstill time for the user to change its behavior. This predictive outcomeis useful for a teacher, parent, school administrator, etc., because itgives them a chance to follow-up with the student to potentially resolvean issue that is preventing the student from completing the assignment.

Continuing the example shown in GUI notification 308, a user (e.g.,teacher) executes a user action 318 selecting a GUI feature associatedwith the predictive insight 316 (predictive outcome). The GUI feature isquestion (“WHY”) prompting the user to select the GUI feature to obtainadditional information. User action 318 is a trigger for display ofprocessing device view 320 (FIG. 3B) subsequently described.

GUI notification 310 provides another example of a predictive outcome asa predicted insight. In the example shown in GUI notification 310, aprediction is made that a plurality of students are predicted to havereduced participation during a future lesson segment (“CreativeWriting”) which is scheduled to being the following week. Thispredictive analysis may have been derived from analysis of mobilitydeterminations that compares students relative to one another withrespect to interactions with a specific topic of a class.

FIG. 3B presents processing device view 320, illustrating a continuedexample of processing from that shown in processing device view 300(FIG. 3A). As previously referenced, processing device view 320 ispresented as a result of receipt of user action 318 selecting a GUIfeature to obtain additional contextual information about a predictiveinsight 316 as shown in FIG. 3A. In response to receipt of user action318, an automatic insight notification 322 is automatically presentedfor the user. Automatic insight notification 322 provides additionaltalking points relative to the presentation of predictive insight 316.For instance, contextual relevance processing of the present disclosureyielded a determination that predicted insight 316 is based oncorrelated data that identifies that a user's group partner has beenabsent the last two days. From correlating user activity data acrossmultiple users, it can be derived that another user's absence isaffecting the performance of a specific student. As a corollary, thepredictive outcome and additional talking points lead to a follow-upnotification 324 being presented that suggests that a teacher follow-upto check on the absent student (“Yossi Amnon”). This correlation is yetanother way in which predictive insights can be used as early warningindications which a teacher can follow-up on before a larger issuepresents itself.

Additionally, the automatic insight notification 322, presented inprocessing device view 320, further comprises a GUI element 326 thatenables a user (e.g., teacher) to dive deeper into this predictiveinsight. For instance, GUI element 326 is presented as a suggestive link(selectable link represented as (“See More”), which is selectable todive deeper into the contextual information. Processing device view 320shows a user action 328 selecting the GUI element 326. User action 328is a trigger for display of processing device view 340 (FIG. 3C)subsequently described.

FIG. 3C presents processing device view 340, illustrating a continuedexample of processing from that shown in processing device view 320(FIG. 3B). As previously referenced, processing device view 340 ispresented based on receipt of user action 328 selecting a GUI feature326 to obtain additional contextual information about a predictiveinsight 316 as shown in FIG. 3A. In response to receipt of user action328, an additional deep level of analysis is provided for the predictiveinsight 316, where automatic insight notification 342 is automaticallypresented for the user. Automatic insight notification 342 presentsadditional talking points that provide rationale diving deeper into howa predictive insight 316 (e.g., predictive outcome) is determined. Inthe example shown, specific data correlations are provided to a user(e.g., teacher) as well as specific types of user activity data that wasdetermined to be relevant to that prediction (e.g., group activity,assignment timeliness from content submissions, user timeliness withrespect to content engagement including reference materials for previousassignments). This correlation is yet another way in which predictiveinsights can be used as early warning indications, and also providingnumerous talking points for a teacher to understand a contextsurrounding a prediction as well as follow-up with students to avoid anegative result.

FIG. 4 illustrates a computing system suitable for implementingprocessing operations described herein related to generation andmanagement of mobility determinations and derivation of predictive datainsights therefrom, with which aspects of the present disclosure may bepracticed. As referenced above, computing system 401 may be configuredto implement processing operations of any component described hereinincluding exemplary components previously described in system diagram100 (FIG. 1 ). As such, computing system 401 may be configured toexecute specific processing operations to solve the technical problemsdescribed herein, which comprise processing operations for intelligentand timely mobility determinations identifying changes that indicate thebeginning of a trend/pattern for which predictive data insights can thenbe generated and presented as early warning indications pertaining touser activity. The idea is that early warning indications, as well ascontextual understanding of the same, enable users to change theirbehavior to affect a predicted outcome before it is too late to so.Computing system 401 may be implemented as a single apparatus, system,or device or may be implemented in a distributed manner as multipleapparatuses, systems, or devices. For example, computing system 401 maycomprise one or more computing devices that execute processing forapplications and/or services over a distributed network to enableexecution of processing operations described herein over one or moreapplications or services. Computing system 401 may comprise a collectionof devices executing processing for front-end applications/services,back-end applications/service or a combination thereof. Computing system401 comprises, but is not limited to, a processing system 402, a storagesystem 403, software 405, communication interface system 407, and userinterface system 409. Processing system 402 is operatively coupled withstorage system 403, communication interface system 407, and userinterface system 409. Non-limiting examples of computer system 401comprise but are not limited to: smart phones, laptops, tablets, PDAs,desktop computers, servers, smart computing devices including televisiondevices and wearable computing devices including VR devices and ARdevices, e-reader devices, gaming consoles and conferencing systems,among other non-limiting examples.

Processing system 402 loads and executes software 405 from storagesystem 403. Software 405 includes one or more software components (e.g.,406A-B) that are configured to enable functionality described herein. Insome examples, computing system 401 may be connected to other computingdevices (e.g., display device, audio devices, servers, mobile/remotedevices, gaming devices, VR devices, AR devices, etc.) to further enableprocessing operations to be executed. When executed by processing system402, software 405 directs processing system 402 to operate as describedherein for at least the various processes, operational scenarios, andsequences discussed in the foregoing implementations. Computing system401 may optionally include additional devices, features, orfunctionality not discussed for purposes of brevity. Computing system401 may further be utilized to execute system diagram 100 (FIG. 1A),process flow 120 (FIG. 1B), process flow 150 (FIG. 1C), processingoperations described in method 200 (FIG. 2 ) and/or the accompanyingdescription of FIGS. 3A-3C.

Referring still to FIG. 4 , processing system 402 may compriseprocessor, a micro-processor and other circuitry that retrieves andexecutes software 405 from storage system 403. Processing system 402 maybe implemented within a single processing device but may also bedistributed across multiple processing devices or sub-systems thatcooperate in executing program instructions. Examples of processingsystem 402 include general purpose central processing units,microprocessors, graphical processing units, application specificprocessors, sound cards, speakers and logic devices, gaming devices, VRdevices, AR devices as well as any other type of processing devices,combinations, or variations thereof.

Storage system 403 may comprise any computer readable storage mediareadable by processing system 402 and capable of storing software 405.Storage system 403 may include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, cache memory or other data. Examples of storage mediainclude random access memory, read only memory, magnetic disks, opticaldisks, flash memory, virtual memory and non-virtual memory, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or other suitable storage media, except for propagatedsignals. In no case is the computer readable storage media a propagatedsignal.

In addition to computer readable storage media, in some implementationsstorage system 403 may also include computer readable communicationmedia over which at least some of software 405 may be communicatedinternally or externally. Storage system 403 may be implemented as asingle storage device but may also be implemented across multiplestorage devices or sub-systems co-located or distributed relative toeach other. Storage system 403 may comprise additional elements, such asa controller, capable of communicating with processing system 402 orpossibly other systems.

Software 405 may be implemented in program instructions and among otherfunctions may, when executed by processing system 402, direct processingsystem 402 to operate as described with respect to the variousoperational scenarios, sequences, and processes illustrated herein. Forexample, software 405 may include program instructions for executing amobility determination management component 406 a; andapplication/service components 406 b, as described herein. In furtherexamples, software may comprise program instructions for executing aseparate component for implementation of a programmed software moduleand/or trained AI processing though in other instances a programmedsoftware module and/or trained AI processing may be executed by one ofthe other components of system diagram 100 in combination with one ormore computing systems 401.

In particular, the program instructions may include various componentsor modules that cooperate or otherwise interact to carry out the variousprocesses and operational scenarios described herein. The variouscomponents or modules may be embodied in compiled or interpretedinstructions, or in some other variation or combination of instructions.The various components or modules may be executed in a synchronous orasynchronous manner, serially or in parallel, in a single threadedenvironment or multi-threaded, or in accordance with any other suitableexecution paradigm, variation, or combination thereof. Software 405 mayinclude additional processes, programs, or components, such as operatingsystem software, virtual machine software, or other applicationsoftware. Software 405 may also comprise firmware or some other form ofmachine-readable processing instructions executable by processing system402.

In general, software 405 may, when loaded into processing system 402 andexecuted, transform a suitable apparatus, system, or device (of whichcomputing system 401 is representative) overall from a general-purposecomputing system into a special-purpose computing system customized toexecute specific processing components described herein as well asprocess data and respond to queries. Indeed, encoding software 405 onstorage system 403 may transform the physical structure of storagesystem 403. The specific transformation of the physical structure maydepend on various factors in different implementations of thisdescription. Examples of such factors may include, but are not limitedto, the technology used to implement the storage media of storage system403 and whether the computer-storage media are characterized as primaryor secondary storage, as well as other factors.

For example, if the computer readable storage media are implemented assemiconductor-based memory, software 405 may transform the physicalstate of the semiconductor memory when the program instructions areencoded therein, such as by transforming the state of transistors,capacitors, or other discrete circuit elements constituting thesemiconductor memory. A similar transformation may occur with respect tomagnetic or optical media. Other transformations of physical media arepossible without departing from the scope of the present description,with the foregoing examples provided only to facilitate the presentdiscussion.

Communication interface system 407 may include communication connectionsand devices that allow for communication with other computing systems(not shown) over communication networks (not shown). Communicationinterface system 407 may also be utilized to cover interfacing betweenprocessing components described herein. Examples of connections anddevices that together allow for inter-system communication may includenetwork interface cards or devices, antennas, satellites, poweramplifiers, RF circuitry, transceivers, and other communicationcircuitry. The connections and devices may communicate overcommunication media to exchange communications with other computingsystems or networks of systems, such as metal, glass, air, or any othersuitable communication media. The aforementioned media, connections, anddevices are well known and need not be discussed at length here.

User interface system 409 is optional and may include a keyboard, amouse, a voice input device, a touch input device for receiving a touchgesture from a user, a motion input device for detecting non-touchgestures and other motions by a user, gaming accessories (e.g.,controllers and/or headsets) and other comparable input devices andassociated processing elements capable of receiving user input from auser. Output devices such as a display, speakers, haptic devices, andother types of output devices may also be included in user interfacesystem 409. In some cases, the input and output devices may be combinedin a single device, such as a display capable of displaying images andreceiving touch gestures. The aforementioned user input and outputdevices are well known in the art and need not be discussed at lengthhere.

User interface system 409 may also include associated user interfacesoftware executable by processing system 402 in support of the varioususer input and output devices discussed above. Separately or inconjunction with each other and other hardware and software elements,the user interface software and user interface devices may support agraphical user interface, a natural user interface, or any other type ofuser interface, for example, that enables front-end processing ofexemplary application/services described herein including rendering of:management of trained AI processing including generation and update oftrained machine learning modeling; management of log data, user activitydata and telemetry data; an improved GUI providing predictive datainsights pertaining to mobility determinations that are used as earlywarning indications; generation and management of contextual datainsights related to predictive data insights derived from mobilitydeterminations; enabling user interactions with GUI elements andfeatures including presentation of GUI menus and callouts, applicationcommand control, etc. and providing notifications through different hostapplication/service endpoints (e.g., via GUI elements, OS notificationsand/or inline with content), among other examples. User interface system409 comprises a graphical user interface that presents graphical userinterface elements representative of any point in the processingdescribed in the foregoing description including processing operationsdescribed in system diagram 100 (FIG. 1A), process flow 120 (FIG. 1B),process flow 150 (FIG. 1C), processing operations described in method200 (FIG. 2 ) and/or the accompanying description of FIGS. 3A-3C.

A graphical user interface of user interface system 409 may further beconfigured to display graphical user interface elements (e.g., datafields, menus, links, graphs, charts, data correlation representationsand identifiers, etc.) that are representations generated fromprocessing described in the foregoing description. Exemplaryapplications/services may further be configured to interface withprocessing components of computing device 401 that enable output ofother types of signals (e.g., audio output, handwritten input, AR/VRinput) in conjunction with operation of exemplary applications/servicesdescribed herein.

Communication between computing system 401 and other computing systems(not shown), may occur over a communication network or networks and inaccordance with various communication protocols, combinations ofprotocols, or variations thereof. Examples include intranets, internets,the Internet, local area networks, wide area networks, wirelessnetworks, wired networks, virtual networks, software defined networks,data center buses, computing backplanes, or any other type of network,combination of network, or variation thereof. The aforementionedcommunication networks and protocols are well known and need not bediscussed at length here. However, some communication protocols that maybe used include, but are not limited to, the Internet protocol (IP,IPv4, IPv6, etc.), the transfer control protocol (TCP), and the userdatagram protocol (UDP), as well as any other suitable communicationprotocol, variation, or combination thereof.

In any of the aforementioned examples in which data, content, or anyother type of information is exchanged, the exchange of information mayoccur in accordance with any of a variety of protocols, including FTP(file transfer protocol), HTTP (hypertext transfer protocol), REST(representational state transfer), WebSocket, DOM (Document ObjectModel), HTML (hypertext markup language), CSS (cascading style sheets),HTML5, XML (extensible markup language), JavaScript, JSON (JavaScriptObject Notation), and AJAX (Asynchronous JavaScript and XML), Bluetooth,infrared, RF, cellular networks, satellite networks, global positioningsystems, as well as any other suitable communication protocol,variation, or combination thereof.

The functional block diagrams, operational scenarios and sequences, andflow diagrams provided in the Figures are representative of exemplarysystems, environments, and methodologies for performing novel aspects ofthe disclosure. While, for purposes of simplicity of explanation,methods included herein may be in the form of a functional diagram,operational scenario or sequence, or flow diagram, and may be describedas a series of acts, it is to be understood and appreciated that themethods are not limited by the order of acts, as some acts may, inaccordance therewith, occur in a different order and/or concurrentlywith other acts from that shown and described herein. For example, thoseskilled in the art will understand and appreciate that a method couldalternatively be represented as a series of interrelated states orevents, such as in a state diagram. Moreover, not all acts illustratedin a methodology may be required for a novel implementation.

The descriptions and figures included herein depict specificimplementations to teach those skilled in the art how to make and usethe best option. For the purpose of teaching inventive principles, someconventional aspects have been simplified or omitted. Those skilled inthe art will appreciate variations from these implementations that fallwithin the scope of the invention. Those skilled in the art will alsoappreciate that the features described above can be combined in variousways to form multiple implementations. As a result, the invention is notlimited to the specific implementations described above, but only by theclaims and their equivalents.

Some non-limiting examples of the present disclosure describe systemsand/or method for managing automated notifications of content updatethrough the generation and presentation of GUI notifications ofpredictive insights. For instance, a computer-implemented method may beexecuted across at least one computing device, including a system and/orcomputer-readable storage media, to accomplish processing describedherein.

A computer-implemented method is implemented that automaticallygenerates predictive insights from analysis of user activity dataassociated with an educational software platform. In doing so, thecomputer-implemented method executes a plurality of processingoperations subsequently described. As a first processing operation, useractivity data is accessed. Exemplary user activity data pertains dataindicating user interactions by a plurality of users with theeducational software platform. The user activity data is raw signal datareceived from a plurality of applications or services associated withthe educational software platform.

Continuing the above example, a trained AI model is automaticallyapplied. The trained AI model is adapted to generate predictive insightsfrom contextual relevance analysis of the user activity data. Thetrained AI model executes numerous processing operations. For instance,the trained AI model generates a plurality of mobility determinationsthat identify changes in patterns of user behavior over a currenttemporal filter associated with the user activity data. The plurality ofmobility determinations are generated based on a collective relevanceanalysis that correlates: data pertaining to current user interactionsof a first user that are identified within the current temporal filter;data pertaining to historical user interactions of the first user thatare identified within a historical temporal filter associated with theuser activity data; data pertaining to current user interactions of oneor more other users that are identified within the current temporalfilter; and data pertaining to historical user interactions of the oneor more other users that are identified within the historical temporalfilter. The trained AI model further curates the plurality of mobilitydeterminations to generate a curated listing of mobility determinationsderived based on an application of business logic rules. The businesslogic rules are configured for an education domain of the educationalsoftware platform and used to evaluate a relevance of the mobilitydeterminations. The trained AI model is further configured to generateone or more of the predictive insights based on an analysis of thecurated listing of the plurality of mobility determinations.

In some examples, the computer-implemented method comprises storing, ona distributed data storage, the one or more predictive insights forrecall. An exemplary GUI notification is generated which isrepresentative of data for rendering the one or more predictiveinsights. Data for rendering the one or more predictive insights is thentransmitted to an application or service of the educational softwareplatform. For instance, the transmitting of the data for rendering theone or more predictive insights retrieves the one or more predictiveinsights from the distributed data storage. The one or more predictiveinsights are able to be rendered in a GUI notification displayablethrough the application or service of the educational software platform.For instance, the computer-implemented method comprises rendering, in aGUI of the application or service, the GUI notification comprising theone or more predictive insights.

In further examples, the generating of the one or more predictiveinsights comprises identifying, from the collective relevance analysisexecuted by the trained AI model, one or more data correlations betweenthe first user and the one or more other users. Additionally, thegenerating of the one or more predictive insights further comprisesincluding the one or more data correlations in the one or morepredictive insights as rationale providing support for a prediction bythe trained AI model. In further technical instances, processingexecuted in generation of the one or more of the predictive insightscomprises: assigning, by application of the trained AI model, aconfidence scoring to the predictive insights based on a correlation,for each of the one or more predictive insights, with one or moremobility determinations included in the curated listing of the pluralityof mobility determinations. The one or more predictive insights aregenerated based on a threshold evaluation of the confidence scoring foreach of the one or more predictive insights.

In additional examples, an evaluation of the relevance of the mobilitydeterminations occurs based on the application of the business logicrules. The application of the business logic rules, executed in thecurating the plurality of mobility determinations, comprises applyingbusiness logic rules that assign a weighting to specific types of useractivity identified within the user activity data. The specific types ofuser activity data are instances of user interactions relative to theeducational domain. The evaluation of the relevance of the mobilitydeterminations comprises generating relevance scoring for each of theplurality of mobility determinations relative to the weighting assigned.The curated listing of mobility determinations comprises one or morecurated mobility determinations identified based on a thresholdevaluation of the relevance scoring for each of the plurality ofmobility determinations.

Reference has been made throughout this specification to “one example”or “an example,” meaning that a particular described feature, structure,or characteristic is included in at least one example. Thus, usage ofsuch phrases may refer to more than just one example. Furthermore, thedescribed features, structures, or characteristics may be combined inany suitable manner in one or more examples.

One skilled in the relevant art may recognize, however, that theexamples may be practiced without one or more of the specific details,or with other methods, resources, materials, etc. In other instances,well known structures, resources, or operations have not been shown ordescribed in detail merely to observe obscuring aspects of the examples.

While sample examples and applications have been illustrated anddescribed, it is to be understood that the examples are not limited tothe precise configuration and resources described above. Variousmodifications, changes, and variations apparent to those skilled in theart may be made in the arrangement, operation, and details of themethods and systems disclosed herein without departing from the scope ofthe claimed examples.

What is claimed is:
 1. A computer-implemented method for automaticallygenerating predictive insights from analysis of user activity dataassociated with an educational software platform, comprising: accessinguser activity data pertaining to user interactions by a plurality ofusers with the educational software platform, wherein the user activitydata is raw signal data received from a plurality of applications orservices associated with the educational software platform;automatically applying a trained artificial intelligence (AI) model thatis adapted to generate predictive insights from contextual relevanceanalysis of the user activity data, wherein the trained AI modelexecutes processing operations that comprise: generating a plurality ofmobility determinations that identify changes in patterns of userbehavior over a current temporal filter associated with the useractivity data, wherein the plurality of mobility determinations aregenerated based on a collective relevance analysis that correlates: datapertaining to current user interactions of a first user that areidentified within the current temporal filter, data pertaining tohistorical user interactions of the first user that are identifiedwithin a historical temporal filter associated with the user activitydata, data pertaining to current user interactions of one or more otherusers that are identified within the current temporal filter, and datapertaining to historical user interactions of the one or more otherusers that are identified within the historical temporal filter, andcurating the plurality of mobility determinations to generate a curatedlisting of mobility determinations derived based on an application ofbusiness logic rules, configured for an education domain of theeducational software platform, that are used to evaluate a relevance ofthe mobility determinations, and generating one or more of thepredictive insights based on an analysis of the curated listing of theplurality of mobility determinations; and transmitting, to anapplication or service of the educational software platform, data forrendering the one or more predictive insights in a graphical userinterface (GUI) notification displayable through the application orservice of the educational software platform.
 2. Thecomputer-implemented method of claim 1, wherein the generating of theone or more predictive insights comprises: identifying, from thecollective relevance analysis executed by the trained AI model, one ormore data correlations between the first user and the one or more otherusers; and including the one or more data correlations in the one ormore predictive insights as rationale providing support for a predictionby the trained AI model.
 3. The computer-implemented method of claim 1,wherein the application of the business logic rules, executed in thecurating the plurality of mobility determinations, comprises applyingbusiness logic rules that assign a weighting to specific types of useractivity identified within the user activity data, and wherein thespecific types of user activity data are instances of user interactionsrelative to the educational domain.
 4. The computer-implemented methodof claim 3, wherein an evaluation of the relevance of the mobilitydeterminations, occurring based on the application of the business logicrules, comprises generating relevance scoring for each of the pluralityof mobility determinations relative to the weighting assigned, andwherein the curated listing of mobility determinations comprises one ormore curated mobility determinations identified based on a thresholdevaluation of the relevance scoring for each of the plurality ofmobility determinations.
 5. The computer-implemented method of claim 1,wherein the generating one or more of the predictive insights based onthe analysis of the curated listing of the plurality of mobilitydeterminations comprises: assigning, by application of the trained AImodel, a confidence scoring to the predictive insights based on acorrelation, for each of the one or more predictive insights, with oneor more mobility determinations included in the curated listing of theplurality of mobility determinations, and wherein the one or morepredictive insights are generated based on a threshold evaluation of theconfidence scoring for each of the one or more predictive insights. 6.The computer-implemented method of claim 1, further comprising: storing,on a distributed data storage, the one or more predictive insights forrecall, and wherein the transmitting of the data for rendering the oneor more predictive insights retrieves the one or more predictiveinsights from the distributed data storage.
 7. The computer-implementedmethod of claim 1, further comprising: generating the GUI notification,and wherein the data for rendering the one or more predictive insightsis data for rendering the GUI notification comprising the one or morepredictive insights.
 8. The computer-implemented method of claim 1,further comprising: rendering, in a GUI of the application or service,the GUI notification comprising the one or more predictive insights. 9.A system comprising: at least one processor; and a memory, operativelyconnected with the at least one processor, storing computer-executableinstructions that, when executed by the at least one processor, causesthe at least one processor to execute a method that comprises: accessinguser activity data pertaining to user interactions by a plurality ofusers with an educational software platform, wherein the user activitydata is raw signal data received from a plurality of applications orservices associated with the educational software platform;automatically applying a trained artificial intelligence (AI) model thatis adapted to generate predictive insights from contextual relevanceanalysis of the user activity data, wherein the trained AI modelexecutes processing operations that comprise: generating a plurality ofmobility determinations that identify changes in patterns of userbehavior over a current temporal filter associated with the useractivity data, wherein the plurality of mobility determinations aregenerated based on a collective relevance analysis that correlates: datapertaining to current user interactions of a first user that areidentified within the current temporal filter, data pertaining tohistorical user interactions of the first user that are identifiedwithin a historical temporal filter associated with the user activitydata, data pertaining to current user interactions of one or more otherusers that are identified within the current temporal filter, and datapertaining to historical user interactions of the one or more otherusers that are identified within the historical temporal filter, andcurating the plurality of mobility determinations to generate a curatedlisting of mobility determinations derived based on an application ofbusiness logic rules, configured for an education domain of theeducational software platform, that are used to evaluate a relevance ofthe mobility determinations, and generating one or more of thepredictive insights based on an analysis of the curated listing of theplurality of mobility determinations; and transmitting, to anapplication or service of the educational software platform, data forrendering the one or more predictive insights in a graphical userinterface (GUI) notification displayable through the application orservice of the educational software platform.
 10. The system of claim 9,wherein the generating of the one or more predictive insights comprises:identifying, from the collective relevance analysis executed by thetrained AI model, one or more data correlations between the first userand the one or more other users; and including the one or more datacorrelations in the one or more predictive insights as rationaleproviding support for a prediction by the trained AI model.
 11. Thesystem of claim 9, wherein the application of the business logic rules,executed in the curating the plurality of mobility determinations,comprises applying business logic rules that assign a weighting tospecific types of user activity identified within the user activitydata, and wherein the specific types of user activity data are instancesof user interactions relative to the educational domain.
 12. The systemof claim 11, wherein an evaluation of the relevance of the mobilitydeterminations, occurring based on the application of the business logicrules, comprises generating relevance scoring for each of the pluralityof mobility determinations relative to the weighting assigned, andwherein the curated listing of mobility determinations comprises one ormore curated mobility determinations identified based on a thresholdevaluation of the relevance scoring for each of the plurality ofmobility determinations.
 13. The system of claim 9, wherein thegenerating one or more of the predictive insights based on the analysisof the curated listing of the plurality of mobility determinationscomprises: assigning, by application of the trained AI model, aconfidence scoring to the predictive insights based on a correlation,for each of the one or more predictive insights, with one or moremobility determinations included in the curated listing of the pluralityof mobility determinations, and wherein the one or more predictiveinsights are generated based on a threshold evaluation of the confidencescoring for each of the one or more predictive insights.
 14. The systemof claim 9, wherein the method, executed by the at least one processor,further comprises: storing, on a distributed data storage, the one ormore predictive insights for recall, and wherein the transmitting of thedata for rendering the one or more predictive insights retrieves the oneor more predictive insights from the distributed data storage.
 15. Thesystem of claim 9, wherein the method, executed by the at least oneprocessor, further comprises: generating the GUI notification, andwherein the data for rendering the one or more predictive insights isdata for rendering the GUI notification comprising the one or morepredictive insights.
 16. The system of claim 9, wherein the method,executed by the at least one processor, further comprises: rendering, ina GUI of the application or service, the GUI notification comprising theone or more predictive insights.
 17. A computer-readable storage mediastoring computer-executable instructions that, when executed by at leastone processor, causes the at least one processor to execute a methodcomprising: accessing user activity data pertaining to user interactionsby a plurality of users with an educational software platform, whereinthe user activity data is raw signal data received from a plurality ofapplications or services associated with the educational softwareplatform; automatically applying a trained artificial intelligence (AI)model that is adapted to generate predictive insights from contextualrelevance analysis of the user activity data, wherein the trained AImodel executes processing operations that comprise: generating aplurality of mobility determinations that identify changes in patternsof user behavior over a current temporal filter associated with the useractivity data, wherein the plurality of mobility determinations aregenerated based on a collective relevance analysis that correlates: datapertaining to current user interactions of a first user that areidentified within the current temporal filter, data pertaining tohistorical user interactions of the first user that are identifiedwithin a historical temporal filter associated with the user activitydata, data pertaining to current user interactions of one or more otherusers that are identified within the current temporal filter, and datapertaining to historical user interactions of the one or more otherusers that are identified within the historical temporal filter, andcurating the plurality of mobility determinations to generate a curatedlisting of mobility determinations derived based on an application ofbusiness logic rules, configured for an education domain of theeducational software platform, that are used to evaluate a relevance ofthe mobility determinations, and generating one or more of thepredictive insights based on an analysis of the curated listing of theplurality of mobility determinations; and transmitting, to anapplication or service of the educational software platform, data forrendering the one or more predictive insights in a graphical userinterface (GUI) notification displayable through the application orservice of the educational software platform.
 18. The computer-readablestorage media of claim 17, wherein the application of the business logicrules, executed in the curating the plurality of mobilitydeterminations, comprises applying business logic rules that assign aweighting to specific types of user activity identified within the useractivity data, and wherein the specific types of user activity data areinstances of user interactions relative to the educational domain. 19.The computer-readable storage media of claim 18, wherein an evaluationof the relevance of the mobility determinations, occurring based on theapplication of the business logic rules, comprises generating relevancescoring for each of the plurality of mobility determinations relative tothe weighting assigned, and wherein the curated listing of mobilitydeterminations comprises one or more curated mobility determinationsidentified based on a threshold evaluation of the relevance scoring foreach of the plurality of mobility determinations.
 20. Thecomputer-readable storage media of claim 17, wherein the generating oneor more of the predictive insights based on the analysis of the curatedlisting of the plurality of mobility determinations comprises:assigning, by application of the trained AI model, a confidence scoringto the predictive insights based on a correlation, for each of the oneor more predictive insights, with one or more mobility determinationsincluded in the curated listing of the plurality of mobilitydeterminations, and wherein the one or more predictive insights aregenerated based on a threshold evaluation of the confidence scoring foreach of the one or more predictive insights.